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ABSTRACT 


ACCESS, A Culture Creating Effective Systems for Success, was a three-year Investing in 
Innovation (i3) development grant funded by the Office of Innovation and Improvement, 
U.S. Department of Education. The ACCESS grant led the way for a districtwide redesign of 
educational practices based on the novel approach of embedding a culture of technology- 
based education (tech culture) within rural schools and the broader community, with an 
intentional effort to record, evaluate, refine, and disseminate an effective model for 
implementation across the state and nation. The impact evaluation included two studies 
that examined the effect of ACCESS on the academic achievement of students in grades 5 - 
10 as measured by NC standardized End-of-Grade assessments and the American College 
Testing (ACT) standardized test. The middle grades study (Study1) used a comparative 
short interrupted time series (CSITS) design to assess the impact of the intervention on 
mathematics and reading achievement in grades 5 through 8 after two years of program 
exposure. The high school study (Study 2) used a quasi-experimental design (QED) to 
assess the impact of ACCESS on ACT composite scores taken in grade 10 after two years of 
program exposure. For both studies, we compared the outcomes of ACCESS participants 
with matched samples of non-participants in other NC school districts. Propensity score 
matching (PSM) was used to match ACCESS samples with comparison students at baseline; 
baseline equivalence was established all pre-test assessment measures. Both studies met 
What Works Clearinghouse (WWC) Group Design Standards with Reservations. The results 
indicated no statistically significant differences in achievement between the ACCESS 
treatment group and the business-as-usual comparison groups as measured by the ACT 
composite scores taken in grade 10 or NC End-of-Grades 5-8 standardized reading and 
mathematics exams. To contextualize the findings, researchers considered the duration of 
students’ exposure to ACCESS, alignment between technology integration and NC 
standardized assessments, and contextual factors that may have narrowed the divide 
between MCS and other school districts from which comparison schools/samples were 
drawn. The report concludes with suggestions for future research and implications for 
education policy. 
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1. INTRODUCTION 


ACCESS, A Culture Creating Effective Systems for Success, was a three-year Investing in 
Innovation (i3) development grant awarded to Montgomery County Schools (MCS) by the 
U.S. Department of Education (U.S. ED). The grant directly addressed U.S. ED’s Priority 6: 
Serving Rural Communities and Priority 5: Effective Use of Technology (Subpart B). ACCESS 
supported a districtwide redesign of educational practices based on the novel approach of 
embedding a culture of technology-based education (tech culture) within schools and the 
community so that students could learn anytime and anywhere in MCS. 


MCS is arrural, hilly, heavily forested area located amid the Uwharrie National Forest and 
Mountain Range, covering 491 square miles with many bus routes for students lasting up 
to two hours. The district qualified under the U.S. Department of Education’s Rural and 
Low-Income School program and serves 4,141 students of whom 74% are economically 
disadvantaged. Additionally, the district has been designated as “low wealth” by the state 
with 21.8% of families living below the federal poverty level (compared to 12.4% 
statewide). The rural location of MCS and its physical geography (heavy forestation, 
mountainous terrain) presented challenges with access to wireless networks and 
consistency in digital connectivity. Additionally, the district experienced challenges 
recruiting and retaining highly-qualified instructors and providing opportunities for 
rigorous and advanced classes. As a districtwide initiative, ACCESS served all MCS students 
and teachers in grades K-12 with an intention to record, evaluate, refine, and disseminate 
an effective model for promoting student outcomes in rural locales across the state and 
nation. 


1.1. Program Description 


ACCESS was a novel approach compared with what had been attempted and studied 
nationally. Earlier initiatives implemented within rural schools included 1:1 device access 
(Butrymowicz, 2012; Kessler, 2011; Morell, 2012), blended and distance learning (Alliance 
for Excellent Education, 2013; Staker & Trotter, 2011), digital learning and assessment 
systems (NCDPI, 2014), mobile study halls (Dillon, 2010), and enhanced networking 
capabilities (Ardery, 2008; Slack, 2013). While these strategies were thoughtful solutions 
that worked to close the gaps in access common to many rural school districts, none of the 
initiatives were designed to redefine the way students learn and how educators teach to 
achieve long-term impact on the field of rural education. The ACCESS model (Figure 1) was 
a systematic attempt to provide solutions to address equity and access to resources for the 
broadest range of students, including those with diverse learning needs, within rural and 
under-resourced school districts (Best & Cohen, 2014; Hannum, Irvin, Banks, & Farmer, 
2009; Jimerson, 2006). 
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Figure 1. The ACCESS Logic Model 


KEY COMPONENTS INDICATORS INTERMEDIATE OUTCOMES 
RESOURCES FOR SUPPORTING HIGH-QUALITY TEACHING AND LEARNING Enhanced Student Engagement: 
e Provide instruction anywhere, anytime via 1:1 Number and percent of: e Increase personal, cognitive, and 
Wi-Fi-based student devices © students with 1:1 devices behavioral engagement 
@ households acquiring discounted home Internet e Decrease in students chronically absent 
service (11-20 days) 
© school buses offering internet access © Decreases in office disciplinary referrals 
e new Wi-Fi “hotspots” added e Decreases in suspensions and 
expulsions 
EXPAND LONG-DISTANCE TEACHING AND LEARNING Enhanced Teacher Efficacy: 
e Expand the opportunities for students to be Number and percent of: e Increase in the percentage of teachers 
taught by effective teachers through distance e distance learning courses offered at MCS high meeting the International Society for 
learning; schools; Technology in Education standards 


e Expand participation in rigorous college courses a 
via the College and Career Promise Program; 
e Expand teacher participation in virtual PLCs; : : : e Increase teacher expectations of students’ 
; ; ¢ e Senior students earning at least one college credit ee 
e Provide professional development to designated jinorea die Cilllgus aaa! Onicer initiee Earnie use and application of technology 
ACCESS Ambassador teachers on technology Co aie tae e Increased levels of communication and 
integration in the classroom; ° teachers participating in virtual PLCs; collaboration among educators 


e Increase in teachers’ self-efficacy at using 


distance learning courses offered at MCS middle 
technology in the classroom 


schools; 


e Provide on-site coaching and integration e ACCESS Ambassador teachers participating in PD; 
support via Technology Facilitators. e Training provided by on-site Technology Facilitators. 
REAL-TIME ASSESSMENT DATA 
e Expand the use of the Learning Management Number and percent of: Improved Student Achievement: 
System (LMS) to coordinate document e teachers utilizing CANVAS LMS system; 
transmission, feeds, sharing of lesson plans, e students reviewing grades on-line with a core teacher at Significant improvement in student 
student work, and access to assessment data in least twice per semester. achievement as evidenced in C-SITS (with 
real time; ES and MS students) and QED (with HS 
e Students review their portfolio on-line 1:1 with students) impact evaluations 


a core course teacher. 
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ACCESS integrated three core components to demonstrate the impact of integrating high- 
quality blended learning technology and training on academic achievement and student 
engagement for rural students in grades K-12. Three core components were implemented 
districtwide: 


1. Distribution and implementation of tech resources; 
2. Expansion of long-distance teaching and learning; and 
3. Expansion of the use of real-time assessment data. 


The following describes the three key components and associated strategies that were 
implemented and evaluated in this study. 


1.1.1 Distribution and implementation of tech resources. 


ACCESS was designed to increased availability and use of educational technology, while 
simultaneously upgrading the density of our wireless network to support that technology 
within and beyond schools. 


Reduced the student to digital learning device ratio. ACCESS enabled MCS to provide 
students with laptops at a 1:1 ratio by increasing the inventory of educational technology 
for grades K-12 (via laptops or tablets). For the duration of the three-year grant, 98% or 
more of eligible MCS students were assigned a laptop or tablet. Exceptions were made for 
students attending MCS alternative school due to the transient nature of the population. 
During Year 1, all students in grades K-12 were assigned a personal 1:1 device the district 
procured with Lenovo. Students in grades 3-12 were able to take their devices home to use; 
our K-2 students used their 1:1 tablet exclusively in the classroom. During the remaining 
years of the grant, all students in grades K-12 continued using the personal 1:1 device that 
was assigned to them by the district. In some of our schools, particularly older schools, the 
bandwidth was below the recommended levels of 10-25 Mbps per 100 concurrent users 
and the number of wireless access points was insufficient to support school-wide 
connectivity. MCS assessed each school’s bandwidth capacity to support upgrades with 
sufficient access points before distributing devices. 


Community hot-spots. ACCESS advanced the installation of free-use wireless (Wi-Fi) “hot- 
spots” throughout Montgomery County. Through partnerships with local businesses, MCS 
established 10 Wi-Fi hot-spots at local restaurants throughout our county which allowed 
rural students and families to have internet access during non-school hours. 


Affordable home-based internet service. MCS leveraged a partnership with a high- 
speed internet service provider to offer qualified MCS families (based on income or 
homebound circumstances) the opportunity to receive home-based internet service at a 
reduced monthly rate; the rate was a substantial reduction from their standard price of 
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$19.95 per month. By the end of 2015, over 60 MCS households exercised this option. By 
the end of 2016, over 70 households were exercising this option. Free home wireless access 
was made available to pregnant or parenting teens for limited time periods via “air cards,” 
sponsored by a host of local businesses. 


Wireless internet service on school buses. ACCESS supported the installation of Wi-Fi on 
select MCS buses with routes longer than 45 minutes and on special education buses to 
enable mobile learning. Through the grant, the district equipped 12 buses with Wi-Fi. Eight 
bus routes within the district had durations of two hours or more. MCS equipped these buses, 
along with four others that serve exceptional children, with Wi-Fi. Wi-Fi on buses allowed 
students online access to complete homework assignments, work on class projects, and 
monitor their grades while in route. 


1.1.2 Expansion of long-distance teaching and learning 


ACCESS expanded opportunities for students to be taught by effective teachers through 
distance learning and facilitated a districtwide redesign of instructional practice to integrate 
work in elementary specialization and multi-classroom leadership with new distance 
learning strategies, allowing MCS access to highly effective teachers via technology. 


Participation in rigorous college courses. ACCESS expanded participation in rigorous 
course work via the College and Career Promise program. The instructional redesign 
facilitated through ACCESS provided every MCS student with the opportunity to take courses 
typically unavailable in rural schools, such as remediation; rigorous and advanced classes 
(Latin, NC Governor’s School for Science and Math); non-traditional subjects and 
opportunities of special interest to our diverse student body (AP Spanish Language and 
Culture); and additional postsecondary courses from a broader range of colleges and 
universities via the Career and College Promise program. 


On-site coaching and integration support. Four digital learning coaches (DLCs; originally 
referred to as instructional technology specialists), were hired and assigned to schools in 
Year 1, one for each middle and high school. Also, our district leveraged Title I funding to 
hire three additional DLCs to support our six elementary schools, each serving two 
elementary schools. In the final year of the grant, four DLCs supported the district, each 
serving an increased number of schools. 


DLCs were certified teachers with instructional technology expertise. Drawing on learning 
adult and student learning theories, they led professional learning development and 
extended that professional development with coaching that included content-specific lesson 
planning focused on effective use of technology, in-class demonstrations of technology, and 
occasional technology troubleshooting. DLCs were able to advance equity in technology use 
within schools and across the district by supporting professional learning communities. 


DLCs received extensive professional development provided by Microsoft, Canvas, 
Nearpod, and The Friday Institute. Each DLC reported growing through the experience, 
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amassing valuable expertise and experience that they used to build the capacity of 
teachers, schools, and the district. Throughout the grant, their roles changed from serving 
as a technician to serving as a learning coach in response to professional development, 
experience, and the evolving needs of teachers. A compilation of advice and best practices 
collected through interviews with DLCs in the final year of the grant appear in Appendix A. 


Technology technicians (one for our middle schools and one for our high schools) were 
hired to provide ongoing assistance with the installation of hardware, diagnosis and repair, 
and maintaining technology to ensure proper functioning in support of student learning 
and teacher professional learning. 


Professional development for ACCESS ambassadors, diplomats, and teachers. Cohorts 
of teachers within each school were designated as ACCESS ambassadors and diplomats. 
ACCESS ambassadors were charged with championing technology integration. Through a 
partnership with The Friday Institute for Educational Innovation (The Friday Institute), 
located within North Carolina State University, a total of 145 teachers from schools across 
the district were recruited and trained to be ACCESS ambassadors. The Friday Institute 
provided four full days of professional development in the use of technology in the 
classroom. A total of 24 teachers completed the diplomat’s course, a supplemental training 
to the ambassador’s program, that was self-paced and focused on a deeper dive into the 
integration of technology into the classroom curriculum. The Friday Institute also provided 
30 hours of professional development for all district administrators. 


Virtual professional learning communities (PLCs) and professional development for 
all teachers. Virtual PLCs were aimed at expanding the effectiveness of high-quality 
teachers across the district. Teachers convened monthly to learn together in support of 
vertical and horizontal alignment across the district. Teachers received technology 
professional development that focused on pedagogy as well as modeling tools that teachers 
could use with their students. Multiple approaches and mechanisms were tried that 
included real-time meetings using videoconferencing technology (limited by varying 
schedules across schools), structured asynchronous discussions, self-paced lessons, and 
asynchronous discussions facilitated by guided practice. By setting up Polycom units for 
each school, teachers collaborated across schools through PLCs. DLCs provided a minimum 
of 10 hours annually of local staff development. 


Additionally, all teachers participated in two Learning Walks each year. Learning Walks 
allowed teachers to visit other classrooms in their school to observe and gather ideas to 
use in their classrooms. The learning walks ranged from general observations to focused 
observations. In the secondary grades, walks occurred during planning time. Between 
observations, led by DLCs, the group debriefed in the hallway. After walks teachers 
submitted reflections to DLCs and often sought each other out to learn more about what 
they observed. MCS also held a summer camp for teachers each year of the grant 
performance period. The summer camp allowed teachers to receive additional training on 
technology and best practices for technology integration. 
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Routine observations and walkthroughs provided data regarding the use of technology and 
pedagogy in the classroom. MCS developed a Technology Integration Walkthrough Tool to 
be used by DLCs and other administrators (see Appendix B). During the time that this was 
implemented within our schools, The Substitution, Augmentation, Modification, and 
Redefinition (SAMR) Model (Puentedura, 2003) level of implementation within classrooms 
increased from substitution and augmentation (within the domain of enhancement) to 
modification and redefinition (within the domain of transformation). In 2016-17, 
Modification and Redefinition were seen in 2.5% of walkthroughs. During 2017-18, this 
percentage increased to 5.5%. 


1.1.3 Expand use of real-time assessment data 


Learning Management Systems (LMS) provides teaching and learning opportunities that are 
highly flexible. Through the use of online platforms students can participate in classes and 
complete coursework at their own pace, offering blended, personalized learning during the 
regular school day or outside of a traditional setting and timeframe. The use of an LMS has 
shown a positive impact on student performance in both national and international studies 
(Martin & Tutty, 2008; Cavus, Uzunbiylu, & Ibrahim, 2006). 


Expand the use of the LMS. ACCESS championed MCS’ Canvas, to coordinate document 
transmission, feeds, sharing of lesson plans, student work, and support real-time teacher and 
student access to assessment data. ACCESS led to meaningful data to continuously improve 
instruction, develop necessary interventions and enrichments, and monitors progress in 
meeting college- and career-ready standards. Portfolios are personalized based on student 
outcome data from district-based assessments provided via a real-time, digital platform 
through Home Base, which in addition to offering instructional content, captures assessment 
information through PowerSchool’s student e-portfolios. 


1.2 ACCESS Fidelity Study 


ACCESS was evaluated on the fidelity of implementation and fidelity of intervention. Fidelity 
of implementation is the extent to which actual project implementation aligned with 
proposed project implementation. Fidelity of intervention advances our understanding of 
quantifiable differences between MCS and business-as-usual conditions in two rural school 
districts from which the Study 2 comparison sample was drawn. 


1.2.1 Fidelity of Implementation 


The ACCESS Annual Fidelity Index, presented in Appendix C, was used to measure fidelity of 
implementation. The index was designed in the first year of implementation. The index is 
comprised of three components (aligned to the three core components), and 11 indicators. 
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The numbers of indicators within each component ranged from two to five and leveraged 
survey data, administrative records, and attendance logs. 


A fidelity score was calculated for each indicator and each construct. The evaluation team 
consulted with the MCS project director and other MCS administrators to distinguish key 
components of the program and the associated activities linked to each component; set 
ambitious, yet achievable, annual thresholds that defined the adequate implementation of 
each activity; and identified suitable data sources to track our progress. The ACCESS lead 
team utilized annual thresholds as benchmarks to assess progress toward their long-term 
program goals and to make course corrections throughout the grant. 


Fidelity data were collected for two school years, SY 2015-16 and SY 2016-17. In SY 2015- 
16, the first year of implementation, two of three ACCESS components were implemented at 
the expected level of fidelity. In Year 2, fidelity of implementation increased, with all three 
ACCESS components being implemented with fidelity. Within Component 2— Expand Long 
Distance Teaching and Learning, access to distance learning courses in MCS middle schools 
(FI 2.2) was a challenge, particularly in Year 2. Within Component 3— Use of Real-time 
Assessment Data, students reporting that they review their progress online with a teacher 
reached the level of implementation expected in Year 2 at one school only. Further study of 
these two components would be helpful to guide future program implementation. 


1.2.2 Fidelity of Intervention 


The NC Digital Learning Progress Rubrics (Friday Institute for Educational Innovation, 2016) 
was used to quantify the differences between MCS and the districts from which the Study 2 
comparison cases were drawn. (Section 4.1 explains how comparison districts were 
selected.) The NC Digital Learning Progress Rubric V2.0 measured district’s implementation 
in five domains: Leadership, Professional Learning, Content and Instruction, Technology and 
Infrastructure, and Data and Assessment. MCS scored highest in all five digital learning 
domains, particularly in the domain of professional learning. However, the differences 
between MCS and comparison district were not as large as expected, particularly in the 
domain of Content and Instruction. See Appendix C for detailed findings. 


2. STUDY 1: IMPACT STUDY DESIGN 


Study 1 used a comparative short interrupted time series (CSITS) design. The eight 
elementary and middle treatment schools were matched to comparison schools ata 1:5 ratio. 
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Schools’ Grades 5-8 reading and math achievement scores were tracked throughout the time 
series, starting five years before the start of the ACCESS intervention and ending two years 
after the start of the intervention. 


2.1 Samples 


The treatment schools consisted of eight rural elementary and middle schools in MCS. There 
were two sets of matched comparison schools (one set for the math analysis and one set for 
the reading analysis). Both sets of comparison schools (math and reading) were drawn from 
two other rural school districts in NC, with five comparison schools matched to each 
treatment school. One set included school-level math achievement scores as a pretest 
covariate, and the other set included school-level reading achievement scores as a pretest 
covariate. Scaled Euclidean distance matching was based on pretreatment school 
characteristics, including percent economically disadvantaged, percent minority students, 
AYP status, baseline standardized test scores, and enrollment. See Appendix D for detailed 
sample demographics. 


The achievement outcomes in this study were grade-within-school test score means. These 


means were derived from test scores from students in Grades 5-8 during the duration of the 
time series. 


2.2 Study 1 Question 
What is the effect of ACCESS on math and reading achievement for ACCESS students after 


three program years compared to the math and reading achievement of students in the 
business as usual condition? 


2.3 Analysis and Results 


2.3.1 Baseline Analytic Model 


M-1 
eae (MatchingBlock,, ) Schs Grades 
ZiGj = Box Bi (Tj) + =! gre ae 


Note that the coefficient B 1 represented the difference between the baseline mean scores of 
treatment and comparison schools. Standardization of the baseline differences was the same 
as described above. 


THE EVALUATION GROUP: PAGE 12 OF 40 


2.3.2 Baseline Analytic Model Specifics 


Using standardized outcomes from the school’s last pre-treatment year only, the following 
model was fit to the combined data set with the standardized outcomes. The model was a 
two-level model with multiple grade-level outcomes (level-1) nested within schools (level- 
2). 


2.3.3 Confirmatory Analytic Model 


M-l 
MatchingB chs rades 
Zigj= Pot P xT) + 8 2 (TreVry) +P 3(T) * TrtYri) + 2 Foon ray ae 75 + 
ge 
where, 

ZiGj = the standardized mean reading/math score from the i time 
point in the Gt grade in the j'® school, (calculation of z-scores 
is described subsequently) 

Tj = 1 if school j is an intervention (treatment) school 

=  Oif comparison school 

TrtYrij = 1 if year is a treatment year 

= 0 if a pre-treatment year 

MatchingBlockm = An indicator variable that takes the value 1 if school was in 
the mt of M matching blocks, and 0 otherwise. 

(Note that there are eight matching blocks, each with one 
treatment school and five comparison schools, so there will 
be seven indicator variables for matching blocks). 

B : = the intercept, which was the comparison school mean score in 
pre-treatment years for schools in the omitted matching 
block. 

B ; = the average difference between treatment and comparison 
schools during pre-treatment years; 

B f = the average difference between pre-treatment years and post- 


treatment years for comparison schools. (Comparison schools 
were never treated, but the “post-treatment years” were the 
years with the treatment schools that they were matched to 
were being treated.) 

B : - The treatment effect. This was the difference-in-difference 
estimator. It was the difference between treatment and 
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comparison schools in their differences between pre- 
treatment and treatment years. 


Pos = was the mt! coefficient for the mt matching block dummy 
variable 

My = the school-specific random intercept 

rGj = _ the grade-specific random intercept 


2.3.4 Analytic Model and Sample Specifics 


The C-SITS design used five years of pre-treatment data and two years of post-treatment 
data. ACCESS was a three-year program, ending 12/31/17. Scores for SY2017-18 were not 
expected to be available until September 2018, or nine months after the end of the project. 
This limited the availability of post-intervention data to two years. 


The impact model was a three-level model with repeated observations over years (level-1) 
nested within grades (level-2), and multiple grades nested in schools (level-3).1 The 
subscripts i, G, and j represented the i* time point for the G'* grade, in the j* school. The 
dependent variable was the standardized mean achievement score for reading or math at 


the ith time point for the Gt grade, in the jt school (standardization discussed below). The 
Schs 
model included random intercepts for schools (denoted as “/  ), random intercepts for 
Grades Years 
grades (denoted as “ _), a residual error term (denoted as ‘‘“ 
variables for matching blocks, and indicator variables for treatment school, treatment years, 


and a treatment school-by-treatment year interaction term. The random terms were each 


), fixed effects dummy 


assumed to be distributed normally with mean zero and variances ers, FErades, and Oyears, 
respectively, and were assumed to be independent of one another. The other model terms 
were as described below. The impact model included an adjustment for potential 
autocorrelation among repeated observations within grades within schools over time. 


In the notation specified above, the betas were model parameters. In the text that follows, 
betas with “hats” are the parameter estimates that were obtained from fitting the model to 


the data. For example, B owas the intercept parameter, and B o was the parameter estimate. 


Because this model was used to estimate and test across-grade impact of ACCESS on reading 
and math achievement (i.e., the combined impact across grades 5-8), we standardized 
achievement test scores so that all grades’ scores were measured via a common metric. 


1 The description of the impact model, and subsequent discussions of standardization and effect sizes, are based 
on Price, C. (February 25, 2015). Example study design plan for a comparative short interrupted time series (C-SITS) 
design: An i3 technical assistance guidance document. 
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During standardization, we used our sample-wide grade-specific school-level means and 
standard deviations from our study sample for each of the school years covered in our study. 
We chose this method to account for our unique sample of rural and high-need target 
schools. We had little or no missing data, as the outcomes were standardized versions of 
schools’ grade-level mean achievement test scores and were publicly available at the North 
Carolina Department of Education website. These mean achievement test scores should have 
only been missing if a school closed or merged with another school. In such circumstances, 
we removed the school from the analysis sample. 


2.3.5 Results for Study 1 


Results indicated no statistically significant difference between the ACCESS treatment group 
and the business-as-usual comparison group on the math achievement outcome (t (1165) = 
-0.275, p = 0.783) or reading achievement outcome (t (1193) = 0.245, p = 0.210). 


The math impact model found baseline scores (t (1165) = 12.246, p < 0.001) and economic 
disadvantage status (EDS, t (1165) = -4.997, p < 0.001) had significant main effects. 
Specifically, higher baseline math scores predicted higher outcome math scores. Moreover, 
EDS qualifying students reported lower outcome scores than students who did not qualify 
for EDS. 


The reading impact model found baseline scores (t (1193) = 9.022, p < 0.001), EDS (t (1193) 
= -7.170, p < 0.001), and minority status (t (1193) = 2.460, p = 0.014) had significant main 
effects. Specifically, higher baseline reading scores were associated with higher outcome 
reading scores. Conversely, FPRL and minority statuses were associated with lower outcome 
reading scores. 


3. STUDY 2: IMPACT STUDY DESIGN 


Study 2 was a quasi-experimental design (QED), focusing on 10‘ graders in the two rural 
MCS high schools and 10* graders in business-as-usual rural high schools in two comparable 
rural school districts. Comparison 10‘ graders were matched to treatment 10* graders 
through a 1:1 ratio using scaled Euclidean distance matching. The cohort of 10‘ graders was 
followed over two years of the ACCESS program, with the confirmatory outcome of ACT 
composite scores assessed in the spring of 2017. 


3.1 Samples 


The treatment schools were the two rural high schools in MCS, and all 10 graders during 
SY2015-16 were included in the study. Comparison schools were drawn from two other 
rural school districts in NC. Because of the unique characteristics of the treatment district, 
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comparison districts were selected based on key demographic variables (percent EDS, 
percent minority, Rural and Low-Income School designation) and in consultation with the 
project team and the district’s assistant superintendent of accountability. A subset of 
comparison districts’ 10 graders in SY2015-16 were matched to treatment 10* graders. 
Nearest neighbor 1:1 propensity score matching was used to match 399 comparison 
students to 224 treatment students. Matching considered pretreatment student 
characteristics of minority status, sex, and baseline math scores. All treatment students in 
Grade 10 were included, as well as comparison students matched to them. See Appendix D 
for detailed sample demographics. 


3.2 Study 2 Questions 


What was the effect of ACCESS on high school students' ACT composite score for ACCESS 
students after two program years compared to the ACT composite scores of high school 
students in the “business as usual” condition? 


3.3 Analysis and Results 


3.3.1 Baseline Analytic Model and Specifics 


All baseline equivalence testing was done on the analysis sample, i.e., a student with both 
pre- and post-test data. No outcome data and no pre-test data was imputed. Baseline 
equivalence was assessed for ACCESS students and comparison students on 9th-grade math 
achievement. The following model was used to estimate the baseline mean difference 
between the intervention and comparison groups. 


BaselineMathScore;; = Bo + TreatmentjB; + pj + 4j 
Where, 
Baseline;; was the continuous outcome (Baseline Math Scores); 
Bo was intercept; 


B, was the difference in baseline math scores for students in the treatment group compared 
to students in the comparison group; 


Treatment, is 1 = treatment group and 0 = comparison group; 
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1; was a school-specific random intercept; and 
e,; was a student-level error term. 


We calculated the standardized baseline mean difference between the ACCESS intervention 
group and the comparison group by dividing the baseline treatment-comparison difference 
(B1 in the level-2 equation) by the student-level pooled standard deviation of pre-test 
mathematics test scores. Given that we included students’ ninth grade baseline mathematics 
score in our impact analysis model, we considered baseline equivalence to be established if 
the standardized mean difference between treatment and comparison on pre-test math 
scores was less than 0.25. 


3.3.2 Analytic Model and Sample Specifics 


Consistent with WWC standards, the analysis sample was defined as cases with non-missing 
outcome and non-missing pre-test data. 


3.3.3. Confirmatory Analytic Model 


Let i index students and j index schools. The following two-level model was used for the 
impact analysis. 
Yij = Bo + Treatment,;B, + BaselineMathScore;;B, + AgejB3 + Gender;;B, 
+ MinorityStatus;B; + 6& + ej 


Where: 

Y;; was the continuous outcome (ACT Composite Score); 

Bo was intercept; 

Treatment; is 1 = treatment student and 0 = comparison student; 


B, was the covariate adjusted impact of ACCESS (the mean difference in ACT composite 
scores for students in the treatment group compared to students in the comparison group); 


BaselineMathScore;; is the baseline math score for each student i; 


B, is the parameter estimate for the effect of student baseline math scores; 


Age; is the age for each student /; 


B3 is the parameter estimate for the effect of student age; 


Gender is 1 = female and 0 = male; 
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GB, is the parameter estimate for the effect of student gender; 

MinorityStatus;; is 1 = minority and 0 = not minority; 

B; is the parameter estimate for the effect of student minority status; 

€; was a school level error term; and 

e;; was a student-level error term. 

3.3.4 Results for Study 2 

Results indicated no statistically significant difference between the ACCESS treatment 
group and the business-as-usual comparison group on the ACT composite score outcome. 
However, baseline math achievement significantly predicted higher ACT composite score 


outcomes. Minority students scored significantly lower on the ACT than non-minority 


students. Table 1 includes our regression model output. 


Table 1. ACCESS ACT Model 


Variable Estimate piandard t-value p-value 
Error 
Intercept 22.91 4.46 5.14 < 0.001 
Mean Centered Baseline Math 0.29 0.02 16.71 < 0.001 
Score 
Treatment -0.19 0.61 -0.32 0.770 
Sex 0.10 0.27 0.37 0.7134 
Minority Status -1.57 0.30 -5.22 < 0.001 
Age -0.33 0.25 -1.31 0.191 
4. DISCUSSION 


ACCESS supported a districtwide redesign of educational practices based on the novel 
approach of embedding a culture of technology-based education (tech culture) within rural 
schools and the community, to close gaps in educational access and promote student 
achievement. The results indicated no statistically significant differences in achievement 
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between the ACCESS treatment group and the business-as-usual comparison groups as 
measured by the ACT composite scores taken in Grade 10 or NC End-of-Grades 5-8 
standardized reading and mathematics exams. To contextualize the findings researchers 
considered the duration of students’ exposure to ACCESS, alignment between technology 
integration and NC standardized assessments, and contextual factors that may have 
narrowed the divide between MCS and other school districts from which comparison 
schools/samples were drawn. 


Instructional redesign of the breadth and scope undertaken by MCS required a substantial 
upfront investment of time in establishing infrastructure, strategically selecting and securing 
technology, staff professional development, and job-embedded coaching based on trusting 
professional relationships. The first year of the grant was foundational with the first half of 
the academic year spent securing and rolling out technology and the second half of the year 
familiarizing teachers and students with new processes, support personnel, curricular 
opportunities, and equipment. By Year 2, high fidelity of implementation study findings 
provided evidence that ACCESS was underway with developing and implementing a high- 
quality redesign of educational practices embedded in a culture of technology integration. In 
Year 3 of the three-year grant, the benefits of the initiative were likely just beginning to take 
hold. Prior research suggests that three to five years may be needed to see the effects of 
major system changes (Borman, Hughes, Overman, and Brown, 2003). Therefore, a longer 
period of time may be needed to detect an effect. 


Furthermore, while the standardized NC state and ACT exams appeared to be well-aligned 
measures to detect the impact of technology integration within the NC standards-based 
curriculum, proximal indicators of teacher and student growth may have been better suited 
to detect shorter-term impacts (less than two years). For example, formative assessments, 
classroom observations (such as the Technology Integration Walkthrough Tool piloted by 
MCS), and indicators of student technology use within and beyond classrooms and schools 
are more Sensitive as early indicators of impact. Additionally, implementation and formative 
evaluation were critical to documenting and detecting shorter-term indicators of impact that 
occur in the early stages of major instructional redesign studies. 


Lastly, another important caveat to consider in interpreting the study findings relates to the 
degree to which the neighboring rural school systems narrowed the divide regarding 
technology-integrated instruction. During the grant period, the NC State Board of Education 
published the NC Digital Learning Plan (September 2015) that emphasized advancing 
education through innovation in digital-age teaching, learning, and leadership. While 
Montgomery County Schools scored higher on the NC Digital Learning Progress Rubric 
overall, and in areas that aligned to the grant, the comparison school districts were not far 
behind (see Appendix C). 
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There were several important lessons that MCS learned through the implementation of 
ACCESS that have both practice and policy implications. 


Plan for device distribution and management. A lesson learned that we implemented as 
we progressed through implementation was to stagger the distribution of devices to students 
and the delivery of professional development to teachers. Before devices are distributed to 
students, teachers require training on basic device management practices, including simple 
troubleshooting techniques, as well as how to develop meaningful content-specific 
curriculum lessons that transform from using technology for technology sake to technology 
for learning sake. An optimal approach would be to provide professional development for a 
period of time before distributing devices to students. Early on some teachers were 
apprehensive about instruction supplemented through laptops and tablets. This may have 
eased the transition by providing teachers time to better prepared and gain comfort with the 
technology. 


In addition to providing training on technology use, pedagogy, and content integration, 
teachers required strategies and tools for 1:1 classroom behavior management. Repeat 
modeling by digital learning coaches of classroom practices proved to be beneficial to our 
teachers. Coaches collected feedback from staff following every training and found that 
professional development delivered by teachers for teachers was beneficial for teacher 
development and morale. Coaches found co-teaching early, often, and with a purpose was 
beneficial to teachers. 


Close attention to distribution, device management, and device maintenance needs to be 
provided to parents and students. Clear and on-going communication for all parties around 
device management practices and procedures are necessary to safeguard and maximize the 
technology investment. A subset of MCS parents worked irregular shift hours, particularly 
those working in the lumber industry, our region’s main employer. The work hours of parents 
made it difficult to gain written permission acknowledging the children’s possession of 
devices and to ensure parents’ support of the proper use of the device. MCS’ solution was to 
schedule several pickup times at each school during different time slots so parents that who 
work different shifts could still come and get their child’s device. 


Community engagement supports technology integration. MCS found that when 
implementing a 1:1 initiative, especially in a rural community, educator, parent, student, and 
community buy-in (or lack) thereof greatly contribute to the adoption of a technology culture. 
Developing and maintaining a trusting and responsive relationships with all stakeholder 
groups - teachers, parents, students, and community members - was essential to moving an 
innovative program forward. All stakeholders needed a good foundational understanding of 
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the importance of a culture shift towards technology use, at the onset of implementing an 
innovative program. 


MCS found success by engaging all stakeholders before receiving funding for ACCESS. Despite 
having early support of stakeholders, district leaders found that continuous surveying and 
communications were essential to obtaining feedback from to monitor progress and refine 
practices. MCS used social media often, utilized the expertise of students, educators, and 
community leaders, as well as reached out to obtain support from other districts with 
expertise in technology integration. 


Before offering professional development opportunities, digital learning coaches found that 
it was essential to conduct a needs assessment of all staff. Coaches reported that being 
intentional about engaging building administrators and curriculum specialists ensured 
technology integration was well aligned with the school’s broader strategic goals. 
Administrative support for technology integration and modeling by school administrators 
encouraged a faster uptake among classroom teachers. 


Some parents and community members expressed concerns about students’ online safety 
with 24/7 internet access. Parent education was required on multiple levels. An extensive 
media campaign was used to raise awareness in the community about what it meant to enable 
students to learn “anytime, anywhere.” MCS publicized ACCESS through the weekly 
newspaper; issued call-outs to families through our Blackboard Connect system; and used 
MCS social media platforms to answer questions and concerns about the program. This 
included demonstrating how to monitor student devices, the benefits of a 1:1 initiative for 
student education, and ways parents can best support students. Parents and students need to 
be empowered with knowledge of good digital citizenship practices. Parents wanted to and 
need to know, upon student’s receiving devices, how to check their child’s browsing history 
and to be supported and encouraged for setting rules for technology use at home. 


Technology integration changes school and classroom climate. MCS teacher-student 
relationships were transformed. DLC and teachers reported that they learned a great deal 
about the abilities of students, even at the kindergarten level. The biggest shifts appeared in 
teachers empowering student self-directed learning and collaboration aided by technology. 
DLCs observed a natural progression in teachers. Instead of business-as-usual lectures, MCS 
teachers were using the technology to facilitate meaningful collaboration (student-student 
and teacher-student), inquiry-based learning, responsible digital citizenship, and meaningful 
feedback using Google tools, Canvas, and other technology. Some teachers even transformed 
their physical classroom spaces. Additionally, teacher collaboration was on the rise. Learning 
walks literally and figuratively opened up classrooms and fostered meaningful exchanges. A 
DLC summarized the journey this way, “ACCESS has given students and teachers a great 
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opportunity to learn and not be behind the curve...Has everything been perfect? No...but 
that’s all part of growing pains and learning...if something doesn’t work, let’s fail forward and 
.. fix it next time.” 


Sustainability planning beyond grant-funding was critical. MCS worked hard to develop 
a plan to sustain ACCESS. The district reallocated local funds and expanded partnerships with 
the private sector. All students in Grades 3 - 12 continued to receive 1:1 devices, although 
only students in Grades 6 - 12 could take devices home. Due to local budget constraints, MCS 
scaled back to having devices available exclusively in each K-2 classroom. In a climate of 
teacher turnover and evolving technology, MCS recognized that DLCs played a critical role in 
pushing the growth curve for teachers, and therefore for students. The district was able to 
locally fund two DLC positions, as well as a fully staffed district technology department. The 
DLC roles solely focus on delivering and fostering professional development within the 
schools, including facilitating learning walks and virtual PLCs. An additional 50 teachers were 
provided ACCESS ambassador training to build capacity within schools. MCS planned to 
increase the number of hotspots available for students to checkout to support the needs of 
rural, high-need students. One of the most important factors influencing sustainability was 
ensuring the continual engagement of teachers, parents, students, and community leaders. 
MCS continued to educate all stakeholders on the necessity for sustaining a 1:1 initiative. 
District and school leaders, as well as digital learning coaches, planned to continue engaging 
stakeholders to keep everyone abreast of programming and emerging technologies that 
advance student development. 


MCS fully implemented ACCESS districtwide and began to have a broader impact. Other 
districts in the state connected to explore adopting and adapting ACCESS. MCS is committed 
to continuing to disseminate the results of our project through media outlets, podcasts, 
websites (district, school, community), social media, publications, and conference 
presentations. MCS is committed to preparing their rural teachers and students to seize the 
opportunities of the next century by placing the world at their fingertips through technology. 
In turn, the district aims to contribute to national policy through continued study and 
dissemination of findings related to technology-based education that expands learning 
opportunities for students in rural locales. 
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APPENDIX A 


Advice and Best Practices Compiled from Interviews with DLCs 


Coaching Teachers for Technology Integration 


Creating a DLC mission statement helped teachers understand the role of coaching. 
It may take a year for teachers to know your safe and not there for evaluative 
purposes. 

Create a schedule that is beneficial to teachers; find ways to integrate PD within 
existing meeting structures like regularly occurring staff meetings in partnership 
with building administrators. 

Need assessments aren’t always fruitful; Developing good rapport and partnerships 
with Instructional Facilitators and Principals to plan professional development 
because they are looking at school data. 

Professional development is not enough; Teachers needed to get comfortable with 
technology and grow in their ability to discern when and when not to use devices to 
support higher levels of learning. 

Twitter and Facebook posts of classroom coaching help to peak interest among 
other teachers. 


Capacity Building for Sustainable Technology Integration 


Work to create lessons that align with the curriculum, even two to four per area, to 
help teachers integrate and move to the next level. 

When customizing professional development to meet the needs of teachers and 
schools, schools might be all over the map. In hindsight, DLC might consider offering 
the same professional development with every school. That way everyone would 
have the same basic knowledge of tools and strategies. 

We should’ve created a repository for all of our old trainings so that when new 
teachers are hired, we do not have to go back and train something we’ve already 
done. 

It is important once you go 1:1 that you Keep that ratio. We are now 

2:1 environment in K-2 and it is hard for teachers to go back. 

We need to be thinking about maintenance. How do we keep this going when this 
grant is over? Teacher spent considerable time integrating and building around the 
technology that has been introduced. 

New teacher will conduct veteran teacher interviews as a virtual PLC activity. New 
teachers can learn from veteran teachers and vice versa. 

Consider creating a troubleshooting list for students and teachers to empower them 
to do some initial diagnosing of challenges (i.e., what to do when my device won’t 
turn on?) 
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As a district, MCS is trying to be more proactive with publishing our work. It is 
important for the district to have a hashtag that everyone can use. It is a great way 
to search published work on social media so other schools learn what’s going on 
throughout the county. 


Student and Parent Engagements 


Students also have to be involved in doing and creating. For example, students can 
create their own online flip cards questions for Kahoot. Sharing the responsible or 
handing off the responsibility to students can be good for teachers and students. 
Have each classroom designate a student as a “tech expert”. 

We should flip our complaints about kids using technology (e.g., talk to text, Facebook, 
etc.) into a teaching strategy. If they are already using these things, how can we use 
them in the classroom? 

Parents need to know everything. Do parent meetings at the beginning of the project, 
each year, and maybe a refresher mid-year to remind parents that their child has a 
device and that they are jointly responsible. 

Do a good job informing parents. We get a lot of pushback from the community and 
parents about the dangers of technology and the internet. Schools need to raise 
awareness about the benefit of technology, as well as safeguards that can be enabled. 


Device Management 


Devices chosen need to be durable with proper cases and teachers must have good 
procedures in place to support device management. 

It’s very important to think about device management (e.g., are barcodes going to 
wear Off or be pulled off). 


It is important for the district to offer insurance for devices because the parts can be 
extremely expensive. 
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APPENDIX B 


Montgomery County Schools Technology Integration Walkthrough Tool 


Montgomery County Schools (NC) developed this tool under an Investing in Innovation grant from the Department of Education 


* Required 


1. Observer * 
Check all that apply. 


2. Grade Level * 
Mark only one oval. 


3. Teacher * 
4. Teacher Email Address: 


5. Student Groupings: (check all observed) * 


Check all that apply. 


Individual 

LC Student Pairs 
C1 Small Groups 
LC] Whole Class 
Other: 


* 


6. What was the Teacher role:(check all observed) 
Check all that apply. 


Lecturing 

L] Discussion 

L] Modeling 

01 Facilitating/Coaching 
1 Interactive Direction 
LO Other: 
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THE 
SAMR 
MODEL 


Dr. Ruben R. Puentedura 


7. Which level of SAMR was evident in the classroom? * 
Mark only one oval. 


Ol Substitution 
OL Augmentation 
O Modification 
OO Redefinition 


8. Describe SAMR level observed: * 


9. Do selected technology tools align with curriculum goals? * 
Mark only one oval. 


Ol yes 

Ol no 

CO inappropriate 

O NA 
10. Do selected technology tools support instructional strategies? * 
Mark only one oval. 

Ol yes 

Ol no 

inappropriate 

O NA 
11. Do selected technology tools support student led learning? * 
Mark only one oval. 

Ol yes 

O no 

O_sinappropriate 

O NA 


12. Suggest Tools * 
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13. Are all students actively engaged in the lesson? Is the teacher monitoring to ensure they are 
on task? * 
Mark only one oval. 


CL] Yes 
[1 No 
Cl N/A 


14. Is the instructional environment conducive to technology-based learning? * 
Mark only one oval. 


Ol Yes 

C1 No 

Oh NA 
15. Is the teacher facilitating learning by allowing student ownership and collaboration? * 
Mark only one oval. 

Ol Yes 

LO) No 

Oh NA 


16. Suggest Improvements * 


17. Teacher uses a web-based platform to distribute materials and foster critical thinking skills? 
(LMS) * 
Mark only one oval. 


CL] Yes 

11 No 

Ol N/A 
18. Teacher communicates with students and colleagues to improve instructional practices? * 
Mark only one oval. 

CL] Yes 

1) No 

LO N/A 
19. Teacher embraces technology, collaborates with colleagues and attends regular trainings to 


implement new tools? * 
Mark only one oval. 


C11 Yes 
[1 No 
Cl N/A 


20. Suggestions * 


21. Communicated with Principal/IF * 


22. PD Response Plan: * 
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APPENDIX C 


ACCESS Annual Fidelity Index and Fidelity of Implementation Findings 


ACCESS was evaluated, in part, on the fidelity of implementation, the extent to which actual 
project implementation aligns with proposed project implementation. The ACCESS Annual 
Fidelity Index was used to measure implementation fidelity. The index is comprised of three 
components and 12 indicators. Indicators, along with measures, targets, and threshold levels 
[Low (L), Moderate (M), and High(H)] for each indicator, were established based on baseline 
data and MCS administrators’ recommendations. Additionally, used to 


Component 1. Resources to Support High-Quality Teaching and Learning 


Fidelity Indicatan &:Measure Year 1 (SY 2015/16) Year 2 (SY 2016/17) 


Indicator Target Actual (A) / Fidelity Score (Level) 


1.1 MCS students with laptops/tablets; Measure: 
Purchase orders A: 100% A: 98% 


2 (H) 2 (H) 
Target: 95% of Students 


1.2 Households acquiring internet service; 
Measure: Century Link Data Orders 


Target: 75 Accounts 
1.3 School buses offering internet access; 
Measure: Administrative records 


Target: 8 School Buses 
1.4 New Wi-fi hot spots; 
Measure: Administrative Records 


Target: 8 School Buses 


Total Program Score 


Program-Level Fidelity Threshold: Score of 6 or more on component score and no individual indicator 
score equals zero. 
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Component 2. Expand Long Distance Teaching and Learning 


Fidelity Indicator & Measure 


Year 1 (SY 2015/16) 


Actual (A) / Fidelity Score (Level) 


Elementary Schools Middle Schools High Schools 
Indicator Target ES1 ES2 ES3 Ast ES4 ESS ES6 MS1 Ms2 HS1 HS2 
2.1 Distance Learning courses at MCS 
high schools (HS Only); Measure: A:4 A:4 
Administrative Records 2 (H) 2 (H) 
Target: 4 courses per HS 
2.2 Distance Learning courses at MCS 
middle schools (MS Only); Measure: A:1 A:1 
Administrative Records 1(M) | 1(M) 
Target: 2 courses per MS 
2.3 Seniors earning at least 1 college 
credit through the College and Career 
Program (CCP; HS Only); Measure: A: 59 A: 35 
Administrative Records 2 (H) 0(L) 
Target: 50% per HS 
2.4 Number of ACCESS Ambassadors (AA) 
by site that attend 4 days AA training; A:6/8 | A:5/6 A: 6/7 A: 4/7 
Measure: Sign-In Sheets; 2 (H) 2 (H) 2 (H) 2 (H) 
Target: 4 AAs attend 4 days 
2.5 Core course teachers participating in 
monthly online PLC sessions 
2.6 Number of training sessions provided 
DLCs by site; Measure: DLC Logs A: 12 A: 15 A: 19 A: 19 A:16 | A:15 A: 12 A: 22 A:15 A: 20 
0 (L) 1 (M) 1 (M) 1(M) | 1(M) | 1(M) | 1(M) 2 (H) 1 (M) 2 (H) 
Target: 20 sessions at all 11 schools 
Total School Score 0 1 1 0 1 1 1 4 5 7 6 


THE EVALUATION GROUP: PAGE 31 OF 40 


Program-Level Fidelity Threshold: 2 of 2 HSs have adequate fidelity (score of 5 or more) AND at least 2 of 2 MSs have adequate fidelity (score of 3 or 
more) AND at least 5 of 6 ESs have adequate fidelity (score of 1 or more); *Data for MLA are reported above, but not included in the final calculation 
of fidelity. 


Actual: 2 of 2 HS; 2 of 2 MS; 5 of 6 ES (Met) 


Year 2 (SY 2016/17) 


Fidelity Indicator & Measure Actual (A) / Fidelity Score (Level) 
Indicator Target Elementary Schools Middle eaisaulilo 
Schools 
HS1 HS2 
2.1 Distance Learning courses at MCS 
high schools (HS Only); Measure: A: 11 
Administrative Records 2 (H) 


Target: 4 courses per HS 


2.2 Distance Learning courses at MCS 
middle schools (MS Only); Measure: A:0 A:0 
Administrative Records O (L) 0 (L) 


Target: 2 courses per MS 


2.3 Seniors earning at least 1 college 
credit through the College and Career 
Program (CCP; HS Only); Measure: 
Administrative Records 


Target: 50% per HS 


2.4 Number of ACCESS Ambassadors (AA) 
by site that attend 4 days AA training; 
Measure: Sign-In Sheets; 


Target: 4 AAs attend 4 days 
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2.5 Core course teachers in Grade 5 and 
up participating in monthly online PLC A: 0 A: 33 A: 100 A: 80 
sessions 0 (L) 0 (L) 2 (H) 2 (H) 


Target: 75% of teachers at all 10 schools 


2.6 Number of training sessions provided 
DLCs by site A: 27 A: 51 A: 42 A:43 | A:27 | A: 44 | A:55 A: 39 A: 53 A: 56 


2(H) ; 2(H) | 2(H) 2(H) ; 2(H) | 2(H) | 2(H) | 2(H) 2 (H) 2 (H) 
Target: 20 sessions at all 11 schools 


Total School Score 4 4 4 3 5 4 4 4 3 8 6 


Program-Level Fidelity Threshold: 2 of 2 HSs have adequate fidelity (score of 5 or more) AND at least 2 of 2 MSs have adequate fidelity (score of 3 or 
more) AND at least 5 of 6 ESs have adequate fidelity (score of 4 or more); *Data for MLA are reported above, but not included in the final calculation 
of fidelity. 

Actual: 2 of 2 HS; 2 of 2 MS; 6 of 6 ES (Met) 
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Component 3. Use of Real-time Assessment Data 


NM all aamssouatere) cer- ale m \Vilelel(-msveatere) sO) \1 mf 


Year 1 (SY 2015/16) Year 2 (SY 2015/16) 
ide lity indicator Measure Actual (A) / Fidelity Score (Level) Actual (A) / Fidelity Score (Level) 
Indicator Target Middle High Middle High 
MS1 MS2 HS1 HS2 MS1 MS2 HS1 HS2 
3.1 Core course teachers using CANVAS at 
least once per month; Measure: ACCESS 
teacher survey A:87 | A:84 | A:93 | A:85 A: 90 A: 87 A: 89 A: 77 
2(H) | 2(H) 2 (H) 2 (H) 2 (H) 2 (H) 2 (H) 2 (H) 
Target: 70% of core course teachers per 
school 
3.2 Students reporting that they review 
their progress online with a teacher at least 
twice per semester; Measure: Tech Talk A:34 | A:43 | A:37 | A:37 A: 35 A: 82 A: 46 A: 40 
enideniciivey o(t) | 1(M) | of) | of) | om | 20) | 1(M) | 1(M) 
Target: 60% of students per school 
Total School Score 2 3 2 2 2 4 3 3 
Program-Level Fidelity Threshold: At least 3 of 4 schools have adequate fidelity (score of 3 or more). 
Actual: 1 of 4 Schools (Not Met) 3 of 4 Schools (Met) 
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NC Digital Learning Progress Rubric 2.0 Score Comparison 
Montgomery County Schools (MCS) and Two Comparison School Districts 
(Friday Institute for Educational Innovation, 2016) 


District Rubric V2.0 MCS Compariso Compariso 
Elements n District 1 n District 2 
Shared Vision 2 1 2 
Personnel 2 1 vi] 
Communication & 
Collaboration 3 2 2 
Sustainability ,) 
LEADERSHIP : : 
Policy 2 2 2 
Continuous 
Improvement 2 1 2 
Procurement 3 2 2 
Overall Score 16 11 14 
Professional 
Development Focus 3 1 2 
Professional 
PROFESSIONAL Development Format 3 2 Zz 
LEARNING Professional 
Development 
Participation 3 1 2 
Overall Score 9 4 6 
Educator Role 2 1 2 
Student-Centered 
Learning 2 2 2 
Access to Digital 
Content 3 3 2 
CONTENTS Ma Prone aa 
ION 
INSTRUCTIO (LMS) , 7 3 
Curation & 
Development 2 1 2 
Data-Informed 
Instruction 2 2 2 
Overall Score 13 12 12 
TECHNOLOGY & School Networks 3 3 3 
INFRASTRUCTURE End-User Devices 2 2 2 
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District Rubric V2.0 
Elements 


Compariso 
n District 1 


Compariso 


hc n District 2 


DATA & 


ASSESSMENT 


13 Learning 
Environments 2 2 2 
T4 Technical Support 3 2 2 
TS Network Services 3 2 2 
T6 Outside of School 3 2 2 
Overall Score 16 13 13 
D1 Data Systems 3 3 2 
D2 Learner Profiles 2 2 2 
Multiple & Varied 
D3 
Assessments 


OVERALL 


RATINGS | 


Overall Score 
Final Score 61 47 51 


| Percent of Possible Points 61% 47% 51% 


Rank P Ne Nee Tarercte Developing Advanced 


Friday Institute Overall Rankings: EARLY (0-25); DEVELOPING (26-50); ADVANCED (51-75); 


TARGET (76-100) 
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APPENDIX D 


Characteristics of the ACCESS Study Analytic Samples 


Table 1. Pre-treatment Years and Treatment Years for Reading and Math Outcomes 
in Grades 5-8 for ACCESS and Comparison Schools 


Spring Spring Spring Spring Spring Spring Spring Number of 


ie 2011 2012 2013 2014 2015 2016 2017 Schools 
ACCESS Schools xX x x x x T T 7 
Comparison Schools =x x xX x x t t 40 
ntime coded as: -4 -3 -2 -1 0 1 2 

TrtYr coded as: 0 0 0 0 0 1 1 


All reading scores come from assessments that were administered in the spring of the school 
year. 

“x”: indicates a pre-treatment year when a school-level grade reading and math outcome 
score was obtained. 

“T”: For Treatment schools, T indicates a treatment year. 

“t’: For comparison schools, “t” indicates a year when the schools’ treatment group 


counterparts received treatment. 


Table 2. Study 1 Baseline Analytics Sample Characteristics 


ASS % Economic 
Daily % Minority = Tie 
Membership peared 
ACCESS Cluster (Math) 370.86 57.49 81.25 
ACCESS Cluster (Reading) 370.86 57.49 81.28 
Comparison Cluster (Math) 362.23 63.79 84.11 
Comparison Cluster (Reading) 370.00 62.17 81.94 
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Table 3. Study 1 Average Grade-Level Achievement Scores by Group and Year 


Reading Mathematics 
Grade by Year of ACCESS Comparison ACCESS Comparison 
Assessment Sample Sample Sample Sample 
Mean Z-Score 
Grade 5 
Spring 2011 (Baseline) 0.22 -0.04 0.33 -0.06 
Spring 2016 (Year 1) 0.28 -0.05 0.26 -0.04 
Spring 2017 (Year 2) -0.14 0.02 0.05 -0.01 
Grade 6 
Spring 2011 (Baseline) 0.21 -0.04 0.31 -0.05 
Spring 2016 (Year 1) 0.26 -0.04 0.25 -0.04 
Spring 2017 (Year 2) -0.09 0.02 0.06 -0.01 
Grade 7 
Spring 2011 (Baseline) 0.31 -0.05 0.37 -0.06 
Spring 2016 (Year 1) 0.13 -0.02 0.20 -0.03 
Spring 2017 (Year 2) -0.06 0.01 0.05 -0.01 
Grade 8 
Spring 2011 (Baseline) 0.10 -0.02 0.25 -0.04 
Spring 2016 (Year 1) -0.01 -0.00 0.21 -0.04 
Spring 2017 (Year 2) -0.20 0.03 0.11 -0.02 
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Table 4. Percent Representativeness of Study 1 Clusters 


Year of Assessment 


Overall 
ACCESS Comparison Analytic 
Clusters Clusters Sample 


Number (and Percentage) of Enrolled Individuals 


Contributing to Cluster Mean 


Study 1 (Elementary Reading) 
Spring 2011 (Baseline) 
Spring 2016 (Year 1) 
Spring 2017 (Year 2) 

Study 1 (Elementary Mathematics) 
Spring 2011 (Baseline) 
Spring 2016 (Year 1) 


Spring 2017 (Year 2) 


2,567 (99) 
2,484 (99) 


2,428 (93) 


2,282 (88) 
2,324 (94) 


2,302 (95) 


15,035 (97) 
15,190 (98) 


15,297 (99) 


9,201 (64) 
9,609 (68) 


9,895 (70) 


17,602 (97) 
17,674 (98) 


17,725 (98) 


11,483 (67) 
11,933 (71) 


12,197 (73) 
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Table 5. Baseline Characteristics of the ACCESS Study 2 Sample 


ACCESS Comparison Standardized 

Characteristics Sample Sample Difference? 
Study 2 (Secondary Achievement): n=224 n=224 

% Gender 68 63 0.13 

% Minority 52 48 0.10 

Standardized Sample Meanon = 252.09 (8.73) 250.07 (7.73) 0.24 

the ACT (SD)> 
NOTES: 


aHedges’ g was used to calculate the standardized differences. Baseline equivalence was 
established if the standardized difference between treatment and comparison groups was 
less than 0.25. Statistical adjustment was included for all key variables. 

bThe standardized baseline mean difference between the ACCESS students and 
comparison/control group students were calculated by dividing the baseline treatment- 
comparison difference by the student-level pooled standard deviation of pre-test scores. 
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